✨ feat: Smooth WRR for Load Balancing#49
Open
zzzhouuu wants to merge 2 commits intoLMRouter:stagingfrom
Open
Conversation
Use the Smooth WRR algorithm to achieve load balancing for providers and API keys.
Contributor
|
Thanks for your contributions! This is indeed very useful feature. I'll review it and merge soon. |
Contributor
|
I'd actually like to make the |
Author
|
Mainly for compatibility and to allow users who don’t need load balancing to configure simply, the API key is now used to store the key with the highest weight for the current provider, making it easier for the underlying code to call. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The current implementation routes all traffic to the first-configured provider when a model has multiple providers; without changing existing logic, introduce the Smooth WRR algorithm to achieve traffic balance among model providers.
Support adding multiple API keys for a single provider and distribute traffic according to their weights.
config example: